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Summary: Digital learning makes 
possible the "unbundling" of school 
provisions— that is, it allows children 
to be served by providers from 
almost anywhere, in new and more 
customized ways. At the same time, 
because it destandardizes and de- 
centralizes educational delivery, 
digital education is far harder to 
bring under the yoke of the quality- 
control systems and metrics that 
have been devised for traditional 
school structures. 

In this paper, Hess explores the pros 
and cons of input regulation, 
outcome-based accountability, and 
market signals as solutions to the 
quality challenge. In the end, he re- 
commends using all three ap- 
proaches in careful combination so 
as to leverage their strengths and 
offset their weaknesses. In practice, 
that means demanding transparent 
financial information from providers, 
holding them to account for student 
achievement gains whenever 
possible, and developing "crowd- 
sourcing" reporting systems to help 
educators, parents, and students 
identify the most effective purveyors 
of online learning. 


Future papers in the series will tackle 
questions surrounding accountability, 
funding, governance, and the edu- 
cator's role in the digital realm. This 
working paper series is generously 
supported by the Charles and Helen 
Schwab Foundation. 



Quality Control in K-12 Digital Learning: 
Three (Imperfect) Approaches 

By Frederick M. Hess 


Digital learning poses an immense 
dilemma when it comes to ensuring 
quality. One of the great advantages 
of online learning is that it makes 
"unbundling" school provision pos- 
sible— that is, it allows children to be 
served by providers from almost 
anywhere, in new and more 
customized ways. But taking ad- 
vantage of all the opportunities 
online learning offers means that 
there is no longer one conventional 
"school" to hold accountable. In- 
stead, students in a given building or 
district may be taking courses (or 
just sections of courses) from a 
variety of providers, each with 
varying approaches to technology, 
instruction, mastery, and so forth. 
(Students may also be benefiting 
from other providers of tutoring, 
out-of-school supplementation, and 
more.) To further complicate this 
picture (and add to its political 
volatility), many providers are likely 
to be profit-seeking ventures. Find- 
ing ways to define, monitor, and 
police quality in this brave new 
world is one of the central chal- 
lenges in realizing the potential of 
digital learning. 

The reformers who shaped our 
current system of schooling in the 


late nineteenth and early twentieth 
centuries drew from the "best prac- 
tices" of their time. In doing so, they 
designed a centrally managed, one- 
size-fits-all system of more-or-less 
standardized schools staffed by 
teachers doing more-or-less stan- 
dardized work. (It was not unlike the 
mass production and assembly line 
systems being devised at approx- 
imately the same time for industry.) 
The presumption was that roughly 
similar schools, school districts, and 
educators could simultaneously 
serve many different students with- 
out altering or tailoring their prac- 
tices in major ways. What may have 
worked 100 years ago, however, has 
now led to overburdened educators 
and institutions that have trouble 
doing anything very well. 

Virtual schooling's greatest power is 
that it creates the opportunity to 
reconsider what is feasible. Digital 
learning makes it possible to deliver 
expertise over distances, permits 
instructors to specialize, allows 
schools to use staff in more targeted 
and cost-effective ways, and cus- 
tomizes the scope, sequence, and 
pacing of curriculum and instruction 
for particular children. All of these 
considerations facilitate the delivery 
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of high-quality, high-impact instruction. At the same 
time, because it destandardizes and decentralizes 
educational delivery, digital education is far harder to 
bring under the yoke of the quality-control systems and 
metrics that have been devised for traditional school 
structures. 

To realize the potential gains in cost efficiency, 
customization, instructional quality, pupil engagement, 
and— ultimately— student learning that the digital age 
makes possible will require policymakers and 
practitioners to find new ways to monitor and police 
quality. Absent the familiar panoply of credentials, 
staffing ratios, instructional hours, Carnegie units, and 
school days that now provide tangible assurance that a 
given school is "real" and legitimate, digital learning will 
struggle with finding acceptance. 

From Quality Schools to Quality Learning 

Embracing the power of digital learning entails shifting 
from a focus on "schools" and "teachers" to one on 
"schooling" and "teaching." Education must no longer be 
understood as something done by holistic, uniform, and 
self-contained organizations but rather as a suite of 
services provided by a shifting web of providers— and 
provided differently in different circumstances to 
different pupils. (That's the point of customizing, after 
all.) Rather than having a faculty that teaches English, 
math, French, and so forth, schools may have multiple 
online providers for each subject— or for portions of a 
particular course. 

For instance, one provider might help students with 
writing and composition, but not teach novels or 
literature. Another might specialize in offering rich, 
interactive instruction about pivotal historical periods, 
without offering a full-scale chronological course. Still 
others (for example, outfits like Rocketship Education or 
Edison Learning) might package content from multiple 
online providers and in-person school faculty to offer 
"blended" instruction. 

Today, most "virtual charter schools" still constitute a 
fairly conventional "school unit," one that can be held 
accountable as a whole for aggregate student 
performance. However, as the unbundling of education 
proceeds, it will no longer be safe to presume that 
student outcomes reflect the performance of a "school" 
or even a "teacher. In an increasingly granular world, 
holding providers responsible for their outcomes requires 
devising ways to gauge the performance of each provider 
in turn, rather than simply documenting the aggregate 


results for children in a given classroom, school unit, or 
locale. Such a task is an enormous challenge, and one far 
beyond the scope of our current abilities, understanding, 
or assessment technologies. While extremely promising, 
then, these unbundled providers offer unique quality- 
control challenges. 

Schooling in a digital world calls upon both school 
personnel and families to make new kinds of choices. 
School leaders will need to decide whether a given course 
should be taught by an online provider, a school-based 
instructor, or some combination of the two— and 
whether to offer students a choice of one or more of 
these modalities. In those cases where more than one 
option is available, students and families will then have to 
make a choice. This new system differs from even our 
current school-choice models, which require only that 
parents decide which school they want John Jr. to attend. 
Instead, it resembles more the relationship between a 
shopkeeper and consumer: The shopkeeper decides 
which goods to put on the shelves, but the consumer 
decides which shop to frequent— and which goods to 
purchase while there. 

Unfortunately, it is difficult to craft quality-control 
systems that reflect and adapt to the seismic shift that 
digital learning represents. The best that policymakers 
can do is to select among— or combine— three basic 
approaches, each with its own significant limitations: 

• Input and process regulation; 

• Outcome-based accountability; and 

• Market-based quality control. 

The alert reader will note that these are precisely the 
same choices available to policymakers seeking to hold 
any public service accountable. 

Input regulation entails policymakers prescribing what 
entities must do to qualify as legitimate online providers. 
Outcome-based accountability relies on setting 
performance targets that providers must meet. And 
market-based quality control permits the universe of 
users to choose their preferred providers— and then 
trusts that market pressures will reward good providers 
and eventually shutter lousy ones. 

These are not mutually exclusive options, but together 
they comprise the basic menu of choices for policing 
digital learning (or any other public function). The 
difficulty is that these approaches were devised for 
assessing conventional institutions, not the more fluid 
networks of providers and learners created by digital 
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instruction. In the digital world — where new tools and 
technologies offer dramatic opportunities to rethink 
teaching and learning by disassembling a school, 
classroom, or course into its component parts, and then 
delivering instruction in more customized ways — these 
quality-control approaches will no longer be a 
comfortable fit for providers. Rather, like a Sunday suit 
that a teen has outgrown, they will tear, pinch, and 
constrict. 

Input Regulation 

Input regulation has long been the norm in policing 
school quality. It entails monitoring and regulating the 
"ingredients" of schooling— who does the teaching, how 
many hours they teach, how many students are taught at 
one time, what materials are used, how much is spent, 
and so forth. This model was superbly suited to the 
circumstances and tastes of the nineteenth and early 
twentieth centuries. A century ago, it was lauded by 
enthusiasts of Frederick Taylor's "scientific manage- 
ment" for its rigor. Monitoring inputs and processes was 
viewed by progressive reformers of the time as the surest 
way to ensure that schools (like factories) were well run 
and efficient. And, in an era before computers and 
modern testing technology, it truly was a reasonable way 
to ensure a baseline of quality. 

Input regulation requires that the school, school district, 
and/or state set and monitor rules detailing how and with 
what resources schools and educators should operate. 
Officials can decree that a classroom will have no more 
than twenty-four students, a teacher will hold a state 
teaching license, new math textbooks will be bought 
every six years, and so on. Rules of this sort ensure a 
minimal level of service, though none of them 
guarantees that students will be served well, much less 
that they will learn. The limitation is that, while officials 
can make sure schools do the things they're told, they 
cannot ensure they do them effectively or well. 
Ultimately, the compliance-oriented approach rewards 
obedience ratherthan excellence. 

Traditional brick-and-mortar schools are relatively easy 
places to deploy this kind of quality control. It isn't hard 
for observers to check up on facilities, materials, and 
staff. A visible student body makes it possible to see 
whether students have books and are showing up on 
time. And a regimented, bureaucratic, and grade- 
ordered school system makes it possible to devise and 
record all manner of input statistics, from attendance 
rates to incidents of violence, that can be used to 
determine whether a given school is "good." 


When it comes to digital learning, input regulation still 
holds considerable appeal, if only because its familiar, 
predictable strictures can mitigate some of the obvious 
risks posed by dubious providers. Skeptics of digital 
learning can be excused for wondering about the invisible 
instructors providing instruction or about the number of 
students each might be teaching. The regulatory 
response is to insist that teachers have certain 
credentials and that class sizes be limited. Similarly, it is 
hardly unreasonable to fear that online courses may be 
too easy or entail little actual instruction. The regulatory 
response is to mandate a minimum number of 
instructional hours and tasks. Input regulation, in other 
words, does offer some protection against fraudulent 
operators. 

Screening out fraudulent or dishonest operators is a low 
bar for ensuring quality, however. The obvious problem 
with input regulation is that policing inputs and processes 
can't ensure quality (much less efficiency). Indeed, its 
rigidities, norms, and compliance obligations tend to 
stifle innovation or efforts to adapt services to the needs 
of schools and students. Nor does traditional regulation 
make much sense in key domains of digital education. 
Consider the issue of teacher quality, for example. If 
online providers presumably have the opportunity to tap 
teachers and professionals around the world, what is 
gained by requiring potential instructors to hold a 
teaching credential from a particular state, a standard to 
which many virtual schools are currently held? Such 
restrictions prevent the use of collegiate faculty or out- 
of-state educators who might be attractive candidates. 
Class size restrictions for digital providers— such as those 
in California — drive up costs while preventing even 
accomplished providers with terrific instructors from 
utilizing their best teachers as effectively as they might. 
(That's every bit as true for providers in brick-and-mortar 
schools, of course.) 

Familiar routines and metrics grow more complex as 
schooling moves online, and they become really tangled 
once a melange of hybrid models starts to operate. 
Blending online and traditional models offers the 
opportunity to reimagine the use of space, resources, and 
staff time, but doing so makes it increasingly 
complicated to devise and apply simple input metrics for 
monitoring quality. 

Regardless of the merits of input regulation in general, 
there is one aspect of online learning for which it is 
indisputably appropriate: monitoring the finances of 
providers that are collecting public dollars. The charter- 
school experience is instructive — and worrying. Nothing 
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will more rapidly (and justifiably) undermine public 
confidence in digital learning than seeing charlatans 
using the medium to collect public dollars. A minimal 
requirement for provision ought to be a clean financial 
audit in which revenues, expenses, and profits are totally 
transparent. (Expect more on this topic from a future 
paper in this series, tentatively titled “Making Our School 
Finance System Work for Digital Learning.") 

Outcome-Based Accountability 

Frustration with input-driven regulation and quality 
control— specifically its rigidities, stifling red tape, and 
inattention to results— gave rise in the 1980s to the 
notion of “reinventing government" by focusing instead 
on outcomes. Public providers would be held 
accountable for results in return for more autonomy, 
flexibility, and control. 

The appeal of this approach is obvious. In K-12 schooling, 
it has meant holding schools accountable for reading and 
math test scores and, more recently, using those results 
to compute value-added measures for schools and 
individual teachers. Test scores offer a crude but useful 
way to identify schools that are doing an awful job 
teaching basic skills, and value-added analyses of those 
test scores yield a method for identifying which schools 
(and potentially individual teachers) are better, and 
which are worse, at helping students master key 
domains. 

The limitation of outcome metrics is that they accept the 
familiar schoolhouse (or classroom) as the unit of 
analysis. They focus on determining whether a school is 
good or not, even if the conventional school no longer 
exists. In cases where a dozen online providers are 
teaching math and the school is functioning more as their 
conduit than as the source of instruction— where learning 
is the result of a blend of school faculty, computer- 
assisted tutoring, and online delivery— school-level 
metrics obviously fail to gauge the relative performance 
of the various providers. 

The type of metrics developed as part of No Child Left 
Behind (NCLB) offer little aid here, for the questions 
needing to be answered involve the quality of particular 
courses, units, or providers, not the performance of the 
school as a whole. T rying to judge the performance of a 
“school" or “teacher" via state assessment results makes 
little sense in the case of digital learning. What's needed 
is something more granular and more reflective of the 
unbundled vision of virtual schooling. 


Conventional approaches to outcome accountability in 
K-12 education unravel when applied to online providers, 
which may teach material that transcends the assessed 
content, or content in disciplines that are not or cannot 
be tested, or only specific portions of an assessed course, 
or skills or capacities for which reliable assessments are 
lacking. For instance, New York City's School of One uses 
dozens of providers to offer components of its middle 
school math curriculum. Different providers offer 
instruction geared to different objectives. Yet the New 
York state assessment measures only how well students 
are faring on grade-level math objectives. It lacks the fine 
granularity necessary to evaluate the individual providers 
of customized content. Similarly, online providers 
offering instruction in music or art history or a foreign 
language— courses that lack meaningful state 

assessments— cannot be readily judged using current 
outcome measures. 

No Child Left Behind's experiment with supplemental 
education services (SES) offers a cautionary tale in this 
regard. SES permitted eligible students in persistently 
low-performing schools to enroll in after-school tutoring 
using federal Title I dollars that had traditionally flowed 
to the school district. Federal policymakers were 
committed to holding SES providers accountable for 
performance. They constructed an elaborate mechanism 
for doing so, with states required to approve potential 
providers and then evaluate their performance based on 
student achievement. Once providers were approved, 
local districts could contract with any number of them, 
and then parents would select the providers of their 
choice. But confusion about how to judge the quality of 
providers, a lack of useful data, and ineffectual state 
efforts to patrol quality combined to yield a potpourri of 
providers of dubious merit. 

Worse, state assessments used for monitoring 
educational outcomes were not precise enough to detect 
the impact of thirty or so hours of tutoring, regardless of 
the provider. This result probably should have come as no 
surprise, but it threw a large wrench into the SES quality- 
control framework. In response, big SES providers asked 
to use their own assessment data to demonstrate their 
impact. But allowing providers to self-police in this 
fashion raised obvious concerns about the integrity and 
reliability of results. 

Still and all, outcome-based accountability should not be 
dismissed altogether where digital learning is concerned. 
On the contrary, new technologies hold immense 
promise for this approach. The development of Common 
Core assessments makes it likely that, within a few years, 
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math and English language arts performance will be 
measured by a uniform test across much of the nation . 1 
Online assessment technology will make it possible to 
administer frequent, targeted tests and get immediate 
results. Such assessments could conceivably be designed 
for every major objective in a set of standards or a 
curriculum, enabling instructors to gauge whether and 
how quickly their pupils are mastering designated 
content. Such assessments could then enable states to 
measure online providers' quality: For online providers 
that augment classroom instruction by, for instance, 
addressing specific thorny concepts, the right gauge is 
not student performance on the end-of-grade 
assessment but on specific learning objectives. Targeted 
testing of this kind would entail a fundamental shift in 
how states have learned to gather and report 
performance data in the NCLB era. 

Tech impresario Tom Vander Ark has suggested a way of 
thinking about student mastery that draws upon the Boy 
Scout concept of the merit badge. Mastery of a given 
learning objective would be signified by completion of a 
designated assessment, and the student would claim the 
appropriate “badge." Vander Ark recommends that seat- 
time credit systems be replaced by assessment bundles 
modeled around these merit badges. “Take ratios and 
fractions as an example," he writes. “A merit badge 
would describe what students need to know and a 
combination of ways they can show it including content- 
embedded assessment (e.g., game score), performance 
assessment (e.g., project), adaptive assessment (e.g., 
online quiz), and an end of unit test ." 2 This doesn't mean 
the assessments will be easy or cheap, however, and 
admittedly such a shift is not likely to occur soon. There 
remain plenty of preliminary challenges to overcome in 
determining which learning objectives ought to be 
identified and what constitutes an appropriate 
assessment. 

Three challenges deserve particular attention: First, for 
the vast majority of courses taught in schools, reliable 
assessments don't even yet exist. Second, we have 
limited experience and expertise in designing 
assessments to track learning objectives rather than 


x The Common Core State Standards (CCSS) are common national 
standards, developed by a consortium of states, in English language 
arts and mathematics. As of July 2011, forty-five states have adopted 
these standards. To supplement them, two state consortia — the 
Partnership for the Assessment of Readiness for College and Careers 
(PARCC) and the SMARTER Balanced Assessment Consortium 
(SBAC) — are developing assessments aligned with the CCSS. 

2 Tom Vander Ark, "Three Pivot Points to a Performance-Based 
Education System," Vander Ark Associates, February 25, 2010, 
http://www.varpartners.net/?p=i545. 


simply report on student performance on the whole of a 
course. Until partial-course metrics are available, 
outcome-based accountability is irrelevant to niche 
providers. Third, most high school assessments report 
whether students are proficient in the subject but not 
how much they learned from the course. This 
characteristic creates incentives for online providers to 
cream-skim, and works against providers that teach 
poorly performing students— since even if these students 
make gains they may not attain proficiency. Devising 
ways to measure both proficiency and value added is 
therefore essential. 

Ultimately, outcome-based quality control for digital 
learning will turn upon the ability of reformers to devise 
and implement a far more complex and sophisticated 
approach to testing. If a student takes Algebra I from one 
provider but absorbs a handful of learning objectives 
from a second, the quality-control challenges are severe. 
First, it's necessary to have an assessment that 
demonstrates a student's mastery of the requisite 
material. Second, it is desirable to have some kind of 
value-added measure that reflects how much the student 
has actually learned. Third, it is important to have 
sufficiently specific items so that the performance of the 
two providers in question can be distinguished. 

Of course, partitioning knowledge and skills into such 
discrete chunks may not finally be possible. (Whether it is 
desirable is another good question.) Even if partitioning 
proves feasible in some subjects or for some instruction, 
it's not clear that it can be done effectively in other 
subjects. If targeted assessments are beyond our 
capabilities to devise, then outcome accountability for 
digital learning will prove elusive. 

In short, the value and reach of outcome-based 
accountability is hostage to the development of high- 
quality, granular assessments. Until new and satisfactory 
assessments are devised, outcome accountability will 
remain a limited tool. And it will remain a nonexistent 
option for all those courses in which assessments do not 
exist and are not yet being developed. 

Market-Based Quality Control 

Market-based accountability can avoid some of the 
problems posed by centralized, outcome-based 
accountability and input-based regulation by permitting 
users of educational services to decide what best meets 
their educational needs. Markets presume that diverse 
users may benefit from diverse providers, and that the 
quality of providers ought to be measured in a variety of 
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ways. Under market-based quality control, the ultimate 
gauge of quality is whether users choose a particular 
provider (thus, the notion of "voting with one's feet"). 
The degree to which competing for students enables 
meaningful quality control, however, depends on the 
degree to which parents and educators are discerning 
and demanding consumers of digital learning. Where 
outcome accountability fosters homogeneity in regard to 
essentials, a market approach creates room for 
heterogeneous providers to thrive. Markets leave room 
for new operators that may not fit within the strictures of 
input regulation and that may offer value or specialized 
services not easily captured by existing outcome 
accountability metrics. 

Ultimately, markets can do three things that input and 
outcome accountability cannot when it comes to quality 
control. First, markets can protect against excessive 
homogenization of providers (and their educational 
products). Second, markets can extend quality control 
beyond the basic forces of regulation and testing. In 
cases where providers might evade regulation or 
manipulate test scores to their advantage, wary 
consumers are free to shift to other providers— along the 
way powerfully signaling their dissatisfaction. (Of course, 
this step depends on consumers having a reasonable 
sense of provider quality and the ability to act on it.) 
Third, markets enable entrepreneurial educators to 
challenge existing schools (and other providers) and the 
reigning orthodoxies and habitual practices. 

Market-based quality control also exhibits particular 
weaknesses, however, including the opportunities it 
offers to inept or unscrupulous providers— a problem that 
is familiar to any education observer who has followed 
the travails of problematic charter schools in states like 
Texas and Ohio. Some educators or parents will make 
poor choices or opt for low-quality providers, maybe 
because they don't know any better, maybe because 
they don't much care, or maybe because they'd rather 
engage with an easy or easy-to-satisfy program than a 
difficult one. It is wishful thinking to suppose that 
markets can prevent bad choices from being made. 
Markets permit consumers to satisfy their preferences, 
including preferences that strike observers as misguided. 
(Consider the millions who eat at fast food joints or buy 
"as-seen-on-TV" gizmos — choices many would deem 
poor ones.) The crucial thing in fostering an effective 
marketplace is that good and reliable information on 
provider quality be available for users. Otherwise, low- 
quality providers can hide their problems, inaccurately 
represent the quality of their service, and use aggressive 
and misleading marketing to woo clients. 


How can markets ensure that users have good 
information on provider quality? The kinds of metrics 
used for outcome accountability can be useful here. But 
four additional kinds of measures can also prove useful in 
this realm: 

• Professional, systematic ratings on customer 
satisfaction, something akin to the kind of 
information reported by sources like J. D. Powers 
and Associates. These make it easy for 
consumers to draw on the judgments of the 
universe of users. 

• Scientific evaluation by credible third-parties, 
such as those offered by Consumer Reports. The 
idea is for experts to put new educational 
products through their paces and then score 
them on relevant dimensions of performance, as 
well as price. 

• Expert evaluation of services like that provided 
by the British School Inspectorate. Unlike input 
accountability, this type of evaluation focuses on 
revealing processes and hard-to-measure 
outcomes. And, unlike the Consumer Reports 
model, such evaluations draw more explicitly on 
informed, subjective judgment and far less on 
laboratory-style experimentation. 

• Data reflecting user experiences— essentially, 
drawing on the wisdom of crowds. TripAdvisor, 
eBay, and Amazon allow the public to readily 
access quality rankings, while also letting users 
offer detailed accounts of their experiences with 
providers. Unlike professional rankings, these 
results do not aspire to be systematic or 
scientific; but for that reason they are especially 
well suited to flagging narrow or particular 
concerns. 

It is not essential that all users access or employ these 
data; it is necessary only that enough do so that others 
can follow their lead, and that providers have reason to 
be responsive. In studying consumer behavior in other 
sectors, economists have suggested that the critical mass 
needed for such a "tipping point" is probably in the 
vicinity of between one-fifth and two-fifths of users. (In 
the case of technology, however, the influence of a small 
group of tech-savvy early adopters is legendary.) 

One other weakness of market-based quality control in 
education rests on the fact that paper credentials, such 
as a certificate of mastery or a high school diploma, have 
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some value— whether or not the recipient has actually 
learned anything. Education providers can satisfy 
customers by helping them acquire degrees or 
certificates that they covet, even if those credentials 
signify no actual educational attainments. This practice is 
particularly common in the "credit recovery" segment of 
the K— 12 digital sector. In higher education, the term 
"diploma mill" has historically referred to institutions that 
provide a diploma in return for little or no demonstrated 
performance. The crude market equivalent is cash for 
diplomas. In most transactions, such a deal wouldn't 
make sense — few consumers would write a check for a 
car that lacks an engine. In education, however, there is 
some value to even a hollow credential. 

Guarding against such chicanery will require some 
combination of external graduation exams (to 
demonstrate that students have learned the requisite 
content), other forms of outcome regulation, and 
potentially input regulation. However, it is again worth 
noting that there are no perfect solutions. The diploma- 
mill problem is one that accreditors have struggled with 
for decades with mixed success. They have tried to 
address it via input requirements that monitor whether 
institutions have the requisite number of books in their 
library, appropriately credentialed faculty, an acceptable 
student-teacher ratio, sufficient seat time, passable 
campus facilities, and so forth. The consequences of such 
regulation are particularly severe for online providers, as 
they force providers to assume a number of unnecessary 
costs or risk being denied accreditation. 

It is not yet clear how savvy educators and parents will 
prove to be in the digital-learning marketplace. They will 
have some opportunity to observe online instruction and 
monitor the outcomes, but the amount that they will be 
able to glean and their vigilance are not yet known (and 
are likely to vary tremendously). Reformers must also be 
wary that any efforts to monitor providers could fall prey 
to a modern-day Horace's Compromise, where students 
are free to coast in return for turning a blind eye to 
mediocre instruction; hard experience teaches that 
youths have typically been quite happy to embrace lax 
instruction if it means less work for them. Because 
participation in markets is ultimately a voluntary 
transaction, the risk that both sides might choose 
mediocrity is real, so long as even unearned credentials 
have value. Policymakers must either decide the risk is 
minimal or devise safeguards that counter or police such 
behaviors (such as insisting upon graduation exams that 
penalize students who have opted for insufficiently 
rigorous instruction). 


Balancing the Three Quality-Control Mechanisms in 
the Digital Era 

Education posed enormous quality-control challenges 
even before the advent of digital learning. Any given 
approach to regulating inputs, basing accountability on 
outcomes, or trusting markets brings risks, 
imperfections, and unintended consequences. Though 
these negatives cannot be eradicated, the alternative- 
no quality control at all — is far worse. So we're well 
advised to recognize and acknowledge the problems with 
available tools and mechanisms and then do our best to 
monitor, minimize, and combat them. 

Regulating inputs like class size, instructional time, and 
staff credentials offers some minimal assurance as to 
what digital providers are actually doing, but carries a 
high cost in terms of stifling potential innovation, 
customization, and cost-efficiency. Policing outcomes 
offers the opportunity to ensure that providers are 
delivering results that meet a given standard for pupil 
growth or achievement, but encourages gamesmanship 
and disputes over the right metrics, even as it deters 
providers whose service doesn't map neatly onto existing 
outcome measures. Markets offer diversity and scope for 
customization, but invite shoddy providers to profit, 
allow some families to be taken advantage of, and 
encourage online providers to focus more on marketing 
than on delivering a high-quality service. 

The risks can be mitigated, if not eliminated, by 
thoughtful design and by combining these approaches 
judiciously. But there is no golden mean or foolproof 
formula. (See sidebar: "How Might Quality Control Work 
in Practice?") Various combinations mostly alleviate 
some concerns by posing new ones. Hence, given our 
scant experience with digital provision, it seems prudent 
to avoid sweeping national policies or requirements, at 
least at this stage. 

While talk of assessment often trends towards the 
hypertechnical, the truth is that accountability and 
quality control are not technical exercises. Instead, they 
require fundamental judgments about how to weigh the 
risks and opportunities posed by mediocrity, red tape, 
standardized outcome measures, and markets. While the 
details of any approach will prove as crucial as they are 
variable, it is possible to discern the outlines of a blended 
approach that seems a sensible way forward for digital 
learning. 

The first step is to create a relatively uncomplicated 
vendor-approval process that ensures minimal fiduciary 
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and academic standards are being met. Providers should 
have to document to a designated public entity that their 
books are clean and to report basic metrics for services 
provided. For those providers that offer certain 
categories of services— especially services that directly 
impact student achievement— it's reasonable to have a 
state review process that features some kind of 
authorization and renewal. 

Second, as providers deliver their wares— and as families 
choose among them and students engage with them— it 
is essential that some institution or institutions collect 
various kinds of data on performance. Data collection is 
apt to be a state responsibility but can easily be 
delegated to any number of third-party monitors, each of 
which would bring its own strengths and weaknesses. But 
whether a state agency acts directly or relies on others, a 
wide array of data needs to be collected, gains measured 
and analyzed, and findings made public in transparent 
fashion. Just as important is to gather and disseminate 
information on consumer satisfaction and expert reviews 
of programs and providers. 

Third, families need to acquire a vested interest in cost- 
effectiveness and new opportunities by being given 
control over some discrete portion of spending. This step 
is essential if parents are to approach schooling as more 
than a unitary service and to start thinking about the 
quality of particular services, and if education officials are 
to enjoy the encouragement and support they need in 
order for them to revisit and change deep-seated 
routines. 

A Word of Caution to Reformers 

In their enthusiasm for virtual learning, reformers risk 
three key mistakes. First, technophiles can too readily 
succumb to the hope that virtual schooling is a rising tide 
that will inevitably sweep away all obstacles before it, 
despite any missteps with regards to quality control or 
incentives. “The power of technology today," write Terry 
Moe and John Chubb in Liberating Learning: Technology, 
Politics, and the Future of American Education, “simply 
cannot be overstated . . . Technology promises to change 
the fundamentals of how teaching and learning have 
taken place for centuries ." 3 In Saving Schools: From 
Horace Mann to Virtual Learning, Paul Peterson suggests 
that “as technology improves, schools can match 
students to their ideal difficulty point, giving them the 


3 Terry M. Moe and John E. Chubb, Liberating Learning: Technology, 
Politics, and the Future of American Education (San Francisco: Jossey- 
Bass, 2009), 71-72. 
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intrinsic satisfaction that comes with a genuine learning 
experience ." 4 

Such sentiments eerily recall the enthusiastic claims once 
made for the transformative power of school choice. 
Twenty years ago, Moe and Chubb asserted that 
“reformers would do well to entertain the notion that 
choice is a panacea ... It has the capacity all by itself to 
bring about the kind of transformation that, for years, 
reformers have been seeking to engineer in myriad other 
ways ." 5 Developments like school choice and digital 
learning do indeed have the potential to be “disruptive" 
forces (in the terminology of Clay Christensen), but there 
is no guarantee that they will play that role . 6 In the case 
of school choice, little or no quality control in too many 
states yielded a slew of mediocre and faltering charter 
schools. Moreover, excessive faith that excellence would 
inevitably win out led reformers to spend little time or 
energy on quality control— yielding middling 
performance and an industry filled with unexceptional 
providers committed to safeguarding their livelihood. 
Such results dampened public enthusiasm and the 
willingness of policymakers to support autonomy and 
opportunities to expand. 

The second key mistake that reformers make is failing to 
acknowledge the implications of public attachment to 
familiar institutions and routines. Proponents of digital 
learning are right to point out that the current system is 
rife with failure. But the system is also familiar, and reaps 
the advantages of popular support, inertia, and the 
benefit of the doubt. Where children and schooling are 
concerned, the burden of proof is going to fall on the new 
and unfamiliar. Worrying about the perils of education 
technology is hardly new. (See sidebar: “Flistoric Tussles 
over Quality in Distance Education.") While today's 
skeptics fret about online instruction, it was once books 
and the printing press that were feared by educators, 
who agonized that students would learn the wrong 
things if left to read on their own. In the seventeenth 
century, Sir Roger L'Estrange (once a member of the 
English Parliament and translator of Aesop's fables) 
wondered “whether more mischief than advantage were 
not occasion'd to the Christian world by the invention of 


4 Paul E. Peterson, Saving Schools: From Fiorace Mann to Virtual 
Learning (Cambridge, MA: Belknap Press of Plarvard University Press, 
2010), 254. 

5 John E. Chubb and Terry M. Moe, Politics, Markets, and America's 
Schools (Washington, D.C.: Brookings Institution Press, 1990), 217. 

6 Clayton Christensen, Michael Plorn, and Curtis W. Johnson, 
Disrupting Class: Flow Disruptive Innovation Will Change the Way the 
World Learns (New York: McGraw-Plill, 2008). 



Historic Tussles over Quality in Distance Education 


Those who imagine that digital learning's challenges are 
unprecedented would do well to consider the 
(surprisingly) long history of distance education. The 
earliest use of distance education in any formal sense is 
probably Isaac Pitman's use of written correspondence to 
teach shorthand as early as 1840 in Bath, England. For 
the cost of a postage stamp, anyone could receive 
shorthand lessons by mail and have those lessons 
corrected and returned in the same fashion. 7 

Experiments with "correspondence schools" emerged as 
early as the 1870s, when formal courses were conducted 
via mail by educational institutions. Their instructors 
were paid by individual students per course. Starting in 
1873, Illinois Wesleyan University experimented with 
distance-instruction degrees. A student could even earn a 
doctorate in philosophy without setting foot on campus. 
The distance learning program received so much 
criticism, though — namely from the University Senate of 
the Methodist Church and from the North Central 
Association of Colleges and Secondary Schools— that the 
program was dead by 1906. 8 Meanwhile, in the 1880s, 
thirty-two professors from universities including Harvard 
and Johns Hopkins formed the Correspondence 
University of America — but it also failed to survive. 

As more for-profit organizations entered the distance- 
education market, peoples' concerns about distance 
schools' practices and the quality of a correspondence 
degree intensified. In those early years, there were two 
noteworthy attempts to control quality. In 1915, the 
growing number of correspondence study options led to 
the formation of the National University Extension 
Association, which sought to establish uniform guidelines 
for distance-learning providers. These guidelines 
included course-transfer procedures and course-quality 
standards. 9 In 1926, a monitoring organization was 
established— the National Home Study Council (NHSC), 


7 Alfred Baker, The Life of Sir Isaac Pitman (Inventor of Phonography) 
(London: I. Pitman & Sons, Ltd., 1908). 

8 William M. French, America's Educational Tradition: An Interpretive 
History ( Boston: D. C. Pleath and Company, 1964), 183. 

9 Proceedings of the National University Extension Conference 

(Princeton, NJ: Princeton University Press, 1915). 


later renamed the Distance Education and Training 
Council (DETC)— to identify high-quality providers in the 
distance-learning field. Today the DETC consists of 
around 100 distance-learning institutions spread across 
twenty-one states and seven countries, including the 
military's distance-learning providers. 

Concerns still abound, however, that these institutions, 
lacking the familiar routines and processes of brick-and- 
mortar colleges, may function as "diploma mills." While 
some traditional institutions may have low standards, it is 
at least evident that they exist. Not even that much can 
be taken for granted about entities that need not 
maintain traditional facilities and cannot boast a visible 
population of faculty and students. Policing the worth 
and meaning of the credentials they offer has proven a 
thorny challenge. 

Seeking to highlight these problems in 2001, U.S. senator 
Susan Collins bought a B.S. in biology and an M.S. in 
medical biology from Lexington University for $1,515, 
without taking any courses. If anyone called to inquire 
about the validity of the degree, the institution assured 
her, it would provide confirmation of her academic 
record and her 3.8 grade-point average. 10 * According to 
former U.S. representative Michael Castle, practices of 
that kind have been "a heck of a lot simpler with the use 
of the Internet. The tracking of them and the prosecution 
of them is a heck of a lot harder." 11 

Those who are confident they have identified sure-fire 
methods for policing the quality of online learning would 
do well to remember that they are not the first to wrestle 
with these issues. Generations of policymakers and 
reformers have sought to juggle the exigencies of quality 
control with the desire not to smother or unduly inhibit 
more convenient, cheaper, or less conventional 
approachesto education. 


10 Dan Carnevale, "Senators Call for a Crackdown on Online Diploma 
Mills," Chronicle of Higher Education, May 21, 2004, A33. 

11 Dan Carnevale, "Taking Aim at Diploma Mills, Education 
Department Creates Online List of Accredited Colleges," Chronicle of 
Higher Education, February 2, 2005, A37. 
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typography ." 12 Newness and unfamiliarity create a high 
bar to clear when assuring parents and the public that 
technology-infused learning (whether it involves books 
or iPads) is not a “risky" departure from what they have 
known. 

Reformers make a third key mistake when they overlook 
the fact that K-12 education is publicly run, funded, and 
regulated, and therefore inherently political. Public 
officials are risk-averse— they want to ensure that public 
dollars and agencies avoid doing obviously corrupt or 
dangerous things. Those opposed to digital learning can 
slow or halt its spread if they can get voters (and public 
officials) worried about the risks involved. It is no surprise 
that union officials and other opponents of digital 
learning are eager to identify and highlight signs of 
malfeasance. When former governors Jeb Bush and Bob 
Wise rolled out their bipartisan Digital Learning Now! 
compact in late 2010, Sherri Wood, president of the 
Idaho Education Association, denounced the digital push 
in the Gem State: “It's about getting a piece of the 
money that goes to public schools. The big corporations 
want to make money off the backs of our children ." 13 
Such attacks tend to gain steam when examples of inept 
or corrupt provision abound, but are less likely to do so 
when problems are minimized. Because these reforms 
are political, success depends in large part on making 
policymakers and the public comfortable with the 
proposed changes. That's where quality control comes in. 

Conclusion 

In the end, creating an authentic and vibrant environ- 
ment for high-quality online learning is about creating a 
new world of schooling that is hospitable to unbundling. 
That world will demand that we make pivotal shifts. 

First, we must begin to think of education in terms of 
segmentation, not just whole schools. Second, we must 
begin to gauge educational quality in terms of cost- 
effectiveness as well as student achievement. Third, we 
must see that parents and educators benefit when they 
make choices that are cost-effective as well as 
educationally beneficial. Finally, we must provide both 
parents and educators with workable, comparable 
metrics by which to gauge both cost and effectiveness. 


12 Sir Roger L'Estrange, "A Rope for Pol, or, A Hue and Cry after 
Marchemont Nedham," (London: [n.p.], 1660). 

13 Trip Gabriel, "More Pupils Are Learning Online, Fueling Debate on 
Quality," New York Times, April 5, 2011, http://www.nytimes.com/ 
20ii/04/o6/education/o6online.html?pagewanted=2&_r=i&sq=virtual 
%2olearning&st=cse&scp=i. 


The challenges involved in effecting these shifts are 
simultaneously familiar and new. In a sense, they are 
essentially the same challenges— to be addressed by the 
same tools— that educators and policymakers have 
wrestled with for decades. But in their current 
incarnation, they can be met only with a degree of 
granularity, agility, and precision that is new to the world 
of K-12 schooling. 

Moving into the digital-learning era, we are burdened by 
our deep-seated habit of regarding schooling as a unitary 
“thing" that happens in “a building" rather than a suite of 
discrete services that can be offered in many locations. 
This creates a tendency to define improvement as 
“better" schools, and to discourage efforts to improve 
discrete services or components. Moreover, schooling in 
America is primarily a public service and thus subject to 
the demands of partisans and interest groups, an 
arrangement that conceals from both parents and 
educators the true costs of goods and services while 
giving them no incentive to emphasize cost- 
effectiveness. These long-standing realities combine to 
stifle new technologies, discourage labor-saving 
improvements, and diminish cost-consciousness. So it's 
no great surprise that technological innovation in 
schooling has consistently disappointed. 

Changing that state of affairs means changing the rules 
that shape schooling. The most significant of these 
govern how schooling itself is provided, paid for, and 
evaluated. Reshaping those rules entails addressing the 
quadruple challenge described above. If the emphasis is 
on learning rather than mere credentialing, and 
especially if the aim is to encourage cost-effective 
learning, it's necessary to relax input regulations in 
exchange for a focus on accountability as measured by 
student outcomes and parental judgements. This step 
means eliminating caps on enrollment, rules that restrict 
class size and student-teacher ratios, geographic and 
regulatory barriers to what online courses students may 
take, and “school site" definitions that limit blended 
models where a portion of student learning occurs 
outside of a traditional school building. It also requires 
clarifying outcome measures by gauging student 
progress based on demonstrated competency or gains 
rather than seat time. 

Transformative improvement also requires stripping 
away conventional rules governing certification and 
licensure so that schools can use unconventional 
personnel in instructional roles, can extend the reach of 
effective teachers, and can tap instructional talent from 
far away. It requires allowing schools to redefine 
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How Might Quality Control Work in Practice? 


Presume that state policymakers wish to aggressively em- 
brace digital learning but, for the reasons discussed earlier, 
are unwilling to leave questions of quality to consumer 
choice or to the market. What then? First, they could stip- 
ulate that for whole-school providers or providers of 
currently assessed courses (i.e., the instruction for which 
conventional measures of performance already exist), ac- 
ceptable performance will be spelled out in terms of student 
outcomes. One challenge is that many state assessments 
may measure student proficiency in a subject without gaug- 
ing student growth, even though the more relevant meas- 
ure of provider quality is growth rather than an absolute 
level of achievement. Unless growth is taken into account, 
there is a huge incentive for digital providers to recruit high- 
achieving students (who will post terrific results) and to shy 
away from low-achievers (who may learn a lot but still not 
be proficient). Just as in the brick-and-mortar world, these 
challenges require the attention to unintended conse- 
quences that has too often eluded policymakers. 

Second, for those providers offering courses or units for 
which such outcomes cannot (currently) be measured, 
policymakers leery of trusting the market have two options. 
One is to put forward a set of input regulations governing 
the time, manner, and nature of provision for at least this 
set of providers. The other is to permit these providers to 
proffer evidence of their own that documents their effec- 
tiveness, though policymakers would need to be confident 
that their education official could distinguish acceptable 


evidence from unacceptable. To help this along, policy- 
makers must opt to set up an input-driven process, but 
agree to provide a waiver from teacher credentials, hours of 
instruction, and class size for those providers who can prove 
to an independently constituted review board that their 
student outcomes pass muster. 

Third, school and district officials would be empowered to 
contract with any provider that has met the state's approval 
criteria. They can opt for whole-course or more specialized 
providers, and can integrate them into their offerings as 
they wish. 

Finally, families would make choices among the offerings, 
both the "schools" and the courses offered by those 
schools. As they do with Advanced Placement or vocational 
education, school officials will inevitably try to steer the 
choices that families make. But families, armed with 
multiple sources of information from online communities 
and philanthropically supported third parties, will ultimately 
choose from the array of options yielded by the state's 
approval process and the decisions of local, charter, and 
virtual school officials. 

A bucolic vision? Certainly not. A messy, flawed system sure 
to be plagued by instances of mediocrity, rigidity, and bad 
decision making? Assuredly. But also a sensible, flexible 
framework that realistically and responsibly mixes and 
matches ourthree approaches. 


instructional roles and staffing, pay, evaluation, and 
scheduling. 

Accountability rooted in student learning requires the 
construction of assessments that measure mastery of 
specific courses or, preferably, of particular units and 
skills. Freeing schools from the long-outdated 
nineteenth-century school calendar involves allowing 
students to take the requisite assessments at any point 
during the year. 

Finally, enabling parents and educators to select indiv- 
idualized online services creates market opportunities for 
providers to specialize in and focus on meeting specific 
needs. Those needs might be for tutoring in specific 
learning objectives or skills or for a complete course of 
instruction. Making such a market viable requires funding 
models that allow fractional per-pupil funds to follow 
students to individual courses, and perhaps even to 
individual instructional units or kinds of support. If dollars 
flow only in chunks that reflect the entire allotment per 
student, or the entire allotment of a particular course, 


then it will be difficult for digital providers to thrive while 
delivering anything that's more finely grained . 14 

Quality control in education, to repeat, is an imperfect 
science, and every approach brings its own short- 
comings. The search for the perfect quality-control 
mechanism is a futile one, just as a laissez-faire disregard 
for quality control is sure to yield practical disappoint- 
ments and political backlash. The sensible course, when 
dealing with a public mission and billions in public funds, 
is to seek an arrangement that addresses concerns about 
malfeasance and mediocrity without stifling innovation — 
and that is able to grow and evolve as we learn and as 
technology and tools mature. A formidable task? Surely; 
because it is one that will ultimately determine whether 
the advent of digital learning revolutionizes American 
education or becomes just another layer of slate strapped 
to the roof of the nineteenth-century schoolhouse. 


14 Expect more on this topic from two future papers in this series, 
tentatively titled "Making Our School Finance System Work for Digital 
Learning" and "The Economics of Online Learning." 
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